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Computing  Environment  for  Adaptive  Multiscale  Simulation 
W91  INF- 12 -1-0371 


Introduction 

This  DURIP  project  provided  funds  for  computing  hardware  to  upgrade  the  computing  capabilities  of 
Rensselaer's  Scientific  Computation  Research  Center  (SCOREC).  The  primary  driver  for  the  requested 
system  at  the  time  of  the  DURIP  proposal  submission  was  our  ongoing  U.S.  Army  Research  Office  grant 
entitled  "Multiscale  Modeling  of  the  Mechanics  of  Advanced  Energetic  Materials  Relevant  to  Detonation 
Prediction",  W911NF-09-1-0330,  Professors  Catalin  R.  Picu  and  Mark  S.  Shephard,  Principal 
Investigators.  As  indicated  below,  this  system  is  being  effectively  applied  to  the  needs  of  that,  and  other 
multiscale  computation  projects.  As  of  September  15,  2014,  this  systems  is  also  supporting  much  of  the 
computational  needs  of  our  new  U.S.  Army  Research  Office  grant  entitled  "Multiscale  Methods  for  the 
Reliable  Simulation  of  Multiphase  Processes",  W911WF-14-1-0301,  Professors  Assad  A.  Oberai,  Onkar 
Sahni  and  Mark  S.  Shephard,  Principal  Investigators.  The  systems  is  also  supports  development  efforts 
on  the  RPI  SCOREC  subcontract  to  an  SBIR  entitled  "Mesh  Generation  and  Control  for  Moving  Boundary 
Problems"  in  support  of  the  Coastal  &  Hyd rau lies  Laboratory  of  the  US  Army  Corps  of  Engineers. 

Computing  System  Upgrades  Support  by  DURIP 

The  funds  from  the  DURIP  grant  supported  the  purchase  of  (i)  a  parallel  computing  cluster,  (ii)  an 
extension  to  the  SCOREC  data  servers  and  storage  system,  (iii)  a  new  network  switch  and  (iv)  high  end 
graphics  workstations. 

Parallel  Compute  Cluster:  The  parallel  compute  cluster  purchased  with  the  DURIP  funds  consists  of  22 
Dell  R620  compute  nodes,  each  with  two  8-core  2.6  GHz  Intel  Xeon  processors,  128  GB  of  DDR3 
1600MHz  RAM,  and  a  direct  connection  to  both  a  56Gbps  Infiniband  fabric  and  a  lOGbps  Ethernet 
fabric.  We  expect  each  node  to  peak  at  about  320  GigaFLOPS,  with  an  aggregate  performance  about  7 
TeraFLOPS.  To  be  able  to  do  larger  problems,  the  22  nodes  have  been  integrated  with  others  purchased 
by  RPI  to  form  a  larger  cluster  totaling  64  nodes  of  the  same  specification,  with  an  aggregate 
performance  of  20  TeraFLOPS.  Although  this  combined  system  is  accessible  to  others,  each  of  the  three 
groups  that  contributed  has  full  access  to  at  least  their  portion  of  the  system  any  time  they  require  it 
and  can  access  more  than  that  portion  at  times  those  nodes  are  not  in  use. 

Data  Servers  and  Stotage:  To  handle  the  increased  I/O  demands,  we  have  added  a  96  terabyte  array 
and  a  14  terabyte  array.  The  96  terabyte  array  handles  most  data  for  the  cluster,  while  the  14  terabyte 
array  is  storing  code,  applications,  and  other  resources  used  by  the  cluster.  Both  arrays  are  built  on  two 
6-core  2.6  GHz  Xeon  Processors,  with  128  gigabytes  of  RAM.  The  96  terabyte  array  consists  of  24  4 
terabyte  SAS  drives  and  the  14  TB  array  consists  of  24  600  gigabyte  10,000  RPM  SAS  disks.  Each  array  is 
capable  of  sustaining  transfer  speeds  of  over  1  Gigabyte/second  and  are  connected  to  a  lOGbps 
Ethernet  Fabric. 
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Network  Switch:  The  10-Gigabit  Ethernet  fabric  is  now  being  supported  by  an  Arista  7150S,  24-port 
Layer-3  switch  along  with  Intel  X520-DA2  network  interface  cards,  which  significantly  increases  our  data 
transfer  throughput  internally.  Additionally,  we  upgraded  our  uplink  to  campus  to  match  the  new 
capabilities  of  the  switch. 

Workstations:  Four  high-end  graphics  workstations  that  greatly  increase  our  ability  to  prepare 
simulations,  develop  code,  and  visualize  results  were  also  added  to  the  SCOREC  system  as  part  of  this 
project.  Each  workstation  has  an  Intel  Core  i7  3.6  GHz  4-core  processor,  32  GB  of  DDR3  1600MHz  RAM, 
and  2GB  Nvidia  GTX580  graphics  card  driving  a  30"  Dell  Ultrasharp  2560x1600  resolution  monitor. 

Application  to  Multiscale  Computations 

The  central  focus  of  our  original  ARO  grant  is  the  simulation  of  the  plasticity  of  the  energetic  material 
RDX.  Figure  1  shows  a  projection  of  the  RDX  crystal  containing  a  dislocation  gliding  on  one  of  the 
horizontal  atomic  planes.  The  defects  of  this  crystal  are  believed  to  play  a  central  role  in  detonation 
initiation.  Detailed  molecular  studies  are  used  to  study  these  mechanisms.  A  typical  simulation  includes 
about  2000  molecules  and  requires  a  time  step  of  2.35E6  fs.  On  the  new  cluster  such  models  require  3 
to  9  hours  each  using  all  352  cores. 


Fig.  1.  [001]  projection  of  a  RDX  crystal.  Fig.  2.  DNA  strand  embedded  in  an  ionic  solution. 

Molecular  dynamics  is  also  used  to  model  the  mechanics  of  DNA  strands  with  full  atomistic  detail.  The 
objective  of  this  work  is  to  quantify  the  strength  of  DNA  junctions;  such  junctions  stabilize  DNA  crystals. 
The  strength  cannot  be  measured  directly  since  manipulation  of  individual  DNA  strands  is  not 
straightforward.  Therefore,  simulations  such  as  the  ones  we  perform  are  the  only  means  to  obtain  order 
of  magnitude  estimates  of  such  parameters.  The  values  are  then  used  to  engineer  DNA  crystals  and  their 


production  process.  Fig.  2  shows  a  DNA  strand  in  an  ionic  solution.  These  simulations  typically  use 
100,000  atoms  and  10,000,000  time  steps  per  stretch  test.  A  single  simulation  requires  24  hours  to 
execute. 

Fibrous  materials  are  ubiquitous  in  everyday  life,  in  engineering  and  biology.  All  such  fiber  networks 
have  some  degree  of  randomness  and  this  aspect  imparts  special  properties  to  the  ensemble. 
Furthermore  most  are  composites  in  the  sense  that  either  or  both  are  made  from  fibers  with  various 
properties  and  embed  inclusions  of  various  sizes  and  shapes.  We  use  large  scale  simulations  to 
represent  the  deformation  and  failure  of  such  structures. 

The  deformation  of  collagen  networks,  relevant  for  tissue  mechanics,  is  modeled  using  large  models  of 
random  fiber  networks.  Models  of  complex  networks  of  polymeric  fibers  such  as  that  shown  in  Fig.  3  are 
developed  and  used  to  represent  components  material  systems.  Other  simplified  networks  such  as 
those  shown  in  Fig.  4,  in  both  2D  and  3D  are  used  for  more  theoretical  purposes  and  to  gain  insights  into 
various  specific  features  of  the  mechanics.  Although  the  individual  simulations  only  require  modest 
parallel  compute  power,  on  the  order  of  500  replicas  are  required  to  be  statically  meaningful. 
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Fig.  3.  Mat  of  polymeric  fibers.  Fig.  4.  2D  models  of  networks  and  their  stress-strain  curves. 

Application  on  New  Projects 

Our  new  ARO  supported  grant  entitled  "Multiscale  Methods  for  the  Reliable  Simulation  of  Multiphase 
Processes"  is  focused  on  the  developing  mathematical  and  computational  modeling  strategies  for 
solving  multiphase  problems  including  ones  involving  chemically  reacting  phases  at  the  interfaces,  and 
problems  with  three  phases  and  contact  lines.  In  order  to  solve  these  problems  we  propose 
thermodynamically  consistent  mathematical  models  and  accompanying  adaptive  discretization 
methods.  Further,  these  problems  require  an  accurate  geometric  representation  of  the  interface 
between  phases  (in  order  to  model  the  chemical  reactions  that  occurs  at  the  interface,  and  to  accurately 
represent  the  curvature  which  is  critical  in  the  contact  line  problem).  Therefore  we  will  develop  novel, 
highly  accurate  interface  tracking  methods  that  will  be  implemented  in  an  infrastructure  that  supports 
the  proposed  adaptive  simulations.  The  DURIP  support  systems  will  be  central  to  the  development, 


9 


testing  and  initial  scaling  of  the  methods  and  associated  software.  The  complete  simulations  will  require 
execution  on  substantially  larger  massively  parallel  systems. 

The  RPI  SCOREC  efforts  on  the  SBIR  entitled  "Mesh  Generation  and  Control  for  Moving  Boundary 
Problems"  are  focused  on  bringing  our  parallel  unstructured  mesh  technologies  to  the  adaptive  solution 
of  large  scale  multiphase  flow  problems  of  importance  to  the  Coastal  &  Hydraulics  Laboratory  of  the  US 
Army  Corps  of  Engineers.  The  SCOREC  team  is  providing  the  parallel  mesh  infrastructure,  parallel  mesh 
adaptation  software  and  is  coordinating  with  researchers  at  the  Coastal  &  Hydraulics  Laboratory  on  the 
development  of  the  discretization  error  estimators  and  adaptive  mesh  control  strategies.  The  DURIP 
systems  will  be  central  to  the  development,  testing  and  initial  scaling  of  the  methods  and  associated 
software  developed  in  this  project. 


